The  views  expressed  in  this  paper  are  those  of  the 
author  and  do  not  necessarily  reflect  the  views  of  the 
Department  of  Defense  or  any  of  its  agencies.  This 
document  may  not  be  released  for  open  publication  until 
it  has  been  cleared  by  the  appropriate  military  service  or 
government  agency. 


STRATEGY 

RESEARCH 

PROJECT 


USE  OF  AN  EXPERT  SYSTEM  IN 
THE  BOARD  SELECTION  PROCESS 


LIEUTENANT  COLONEL  REBECCA  S.  STOREY 
United  States  Army 


360603  261 


DISTRIBUTION  STATEMENT  A: 
Approved  for  public  release. 
Distribution  is  unlimited 


USAWC  CLASS  OF  1996 


U.S.  ARMY  WAR  COLLEGE.  CARLISLE  BARRACKS.  PA  17013-5050 


USAWC  STRATEGY  RESEARCH  PROJECT 


The  views  expressed  in  this  paper  are  those  of 
the  author  and  do  not  necessarily  reflect  the 
views  of  the  Department  of  Defense  or  any  of 
its  agencies.  This  document  may  not  be 
released  for  open  piiblication  until  it  has 
been  cleared  by  the  appropriate  military 
service  or  government  agency. 


Use  of  an  Expert  System  in  the  Board  Selection  Process 


by 

Lieutenant  Colonel  Rebecca  S.  Storey 
United  States  Army 

Dr.  Jay  Liebowitz 
Project  Adviser 


DISTRIBUTION  STATEMENT  A:  Approved  for  public 
release.  Distribution  is  unlimited. 

U.S.  Army  War  College 
Carlisle  Barracks,  Pennsylvania  17013 


ABSTRACT 


AUTHOR:  Rebecca  S.  Storey  (LTC),  USA 

TITLE:  Use  of  an  Expert  System  in  the  Board  Selection  Process 

FORMAT:  Strategic  Research  Project 

DATE:  15  April  1996  PAGES:  20  CLASSIFICATION:  Unclassified 

The  purpose  of  this  study  was  to  begin  the  development  and  testing  of  an  Expert 
System  (ES)  to  screen  officer  personnel  records  being  considered  for  Command  and  General 
Staff  College  (CGSC).  The  project  included  a  review  of  the  artificial  intelligence  (AI) 
literature  relevant  to  the  military  and  hmnan  resource  management,  problem  selection, 
knowledge  acquisition,  knowledge  representation,  knowledge  encoding,  and  knowledge 
testing  and  evaluation.  The  AI  computer  language  PROLOG  was  used  to  develop  a  basic 
rule-based  expert  system.  Data  from  15  mock  officer  personnel  records  obtained  from  US 
Army  Total  Personnel  Command  (PERSCOM)  were  entered  into  Board  Expert  (BOARDEX). 
Results  from  BOARDEX  were  correlated  to  votes  generated  by  a  board  consisting  of  ten 
human  experts  from  the  US  Army  War  College  Class  of  1996.  Numerous  statistically 
significant  correlations  were  found  between  the  members  of  the  board  as  well  as  between  the 
human  experts  and  BOARDEX  (r  =  .6614,  p  =  .007). 
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Use  of  an  Expert  System  in  the  Board  Selection  Plrocess 


Context  of  the  Stu(fy 

The  United  States  Army  has  recently  completed  roughly  95  percent  of  the  largest 
manpower  draw  down  in  its  history.  As  the  Army  becomes  smaller,  some  functional  areas 
change,  some  are  transferred  to  other  agencies,  and  some  simply  disappear.  Specific 
administrative  and  human  resource  oriented  functional  areas  will  of  necessity  remain  intact  in 
order  to  maintain,  train,  and  educate  the  force.  One  of  these  administrative  essentials  is  the 
review  of  officer  records  by  a  school  selection  board.  This  activity  is  of  particular  interest 
during  a  time  of  manpower  reductions  due  to  the  number  of  experts  required  to  convene  a 
duly  constituted  board  to  select  candidates  to  attend  military  schools.  Can  the  Army  afford  to 
continue  such  labor  intensive  processes  in  the  face  of  shrinking  budgets  and  a  smaller  force? 
Can  a  shrinking  manpower  pool  afford  to  permit  a  general  officer  and  at  least  nine  other 
Lieutenant  Colonels  and  Colonels  to  sit  annually  on  a  number  of  promotion,  school  selection, 
and  command  boards  each  year  for  a  minimum  of  five  days  each? 

Completion  of  Command  and  General  Staff  College  (CGSC)  is  one  of  the  imperative 
elements  in  an  officer's  education,  training,  and  career  development.  This  course  is  designed 
to  educate  and  train  field  grade  officers  (the  military  equivalent  of  "middle  management")  in 
the  operational  art  of  warfare.  It  prepares  officers  to  assume  positions  of  leadership  in  the 
military.  Selection  to  attend  CGSC  in  residence  is  a  sought-after  assignment  and  represents  a 
distinct  accolade  for  those  officers  intent  upon  making  the  Army  a  career.  Resident 
enrollment  at  CGSC  is  limited  therefore  a  selection  process  is  required  to  choose  specific 
individuals  who,  based  upon  their  records,  merit  the  opportunity  to  attend  the  ten  month 
course.  The  best  qualified  individuals  are  chosen  from  a  candidate  pool  which  is  derived 


from  a  window  of  specific  basic  year  groups.  CGSC  can  be  done  using  the  corresponding 
studies  option,  however,  the  value  and  prestige  of  attending  the  Army's  CGSC  or  other 
Military  Education  Level  4  (MEL  4)  school  is  undisputed  among  the  officer  rank  and  file  as 
it  signifies  the  trust  and  confidence  that  a  selection  board  has  regarding  the  capabilities  of 
those  chosen. 

The  procedures  and  composition  of  the  CGSC  selection  board  are  defined  by  Deputy 
Chief  of  Staff  for  Personnel  (DCSPER)  Officer  Selection  Board  Support  Standard  Operating 
Procedures  Army  Regulation  351-1  (Individual  Military  Education  and  Training)^, 
Department  of  the  Army  Pamphlet  600-3  (Commissioned  Officer  Professional  Development 
and  Utilization)^,  Officer  Personnel  Management  Division  Operating  Instruction  351-1(4) 
(Command  and  General  Staff  College  Selection  and  Slating  System)^,  and  annual  board 
guidance.  Records  of  officers  in  the  Army  Competitive  Categories  (ACC)  are  boarded  at  one 
time  while  the  records  of  officers  in  the  special  branches  (Army  Medical  Department 
(AMEDD),  Judge  Advocate  General  Corps  (JAG),  and  Chaplains  Corps)  are  boarded 
separately.  The  Special  Branch  records  are  reviewed  by  a  panel  comprised  of  officers  who 
represent  not  only  the  special  branch  under  consideration,  but  also  representatives  of  the 
ACC  as  well. 

The  board  process  for  CGSC  selection  consists  of  two  distinct  elements.  First,  all 
records  are  screened  by  the  entire  board  and  simply  voted  as  Yes  or  No.  In  general,  a  Yes 
vote  means  that  the  officer  is  fully  qualified  for  consideration;  a  No  vote  indicates  that  the 
reviewing  board  member  finds  that  officer  not  qualified  to  be  considered  for  CGSC.  The 
amount  of  time  that  this  first  cut  of  records  takes  is  dependent  upon  the  number  of  records  in 
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the  pool,  but  an  average  length  of  time  that  is  recommended  to  review  each  record  during 
this  phase  is  less  than  three  minutes. 

During  my  personal  experience  on  a  selection  board,  the  board  president  suggested 
that  no  more  than  90  seconds  be  spent  considering  each  file.  Board  members  were  expected 
to  review  a  photograph,  a  hard  copy  of  the  Officer  Record  Brief  (ORB),  a  hard  copy  of  the 
most  recent  Officer  Efficiency  Report  (OER),  and  a  companion  microfiche  (which  consists  of 
microfilmed  copies  of  all  other  existing  OER's,  letters  of  commendation  and/or  reprimand, 
and  copies  of  all  award  certificates)  in  a  very  short  period  of  time.  The  ORB  alone  has  11 
detailed  sections  with  numerous  subordinate  fields  of  data  to  be  visually  scanned  by  each 
board  member. 

The  second  portion  of  the  CGSC  board  procedure  is  the  hard  vote.  After  each  record 
has  been  reviewed  by  every  board  member,  a  report  is  generated  which  tallies  the  number  of 
Yes  and  No  votes  per  record.  In  general,  a  record  will  proceed  to  the  hard  vote  portion  of 
the  board  process  if  at  least  one-half  of  the  board  members  have  voted  Yes  on  it.  No 
records  are  dropped  from  consideration  at  this  point.  Board  members  then  proceed  by  rating 
the  remaining  records  on  a  scale  of  1  through  6.  Pluses  and  minuses  may  be  included  at  this 
time  to  differentiate  between  similar  records. 

Once  the  hard  vote  is  complete,  the  board  secretary  provides  the  board  members  with 
an  order  of  merit  list  (OML).  The  OML  is  compiled  by  summing  the  numeric  votes  from 
the  board  members  for  each  individual  record.  The  officer  receiving  the  highest  vote  count 
is  at  the  top  of  the  list,  the  person  receiving  the  next  highest  total  is  second,  and  so  on. 

Pluses  and  minuses  are  reflected  on  the  OML  lists  and  contribute  to  the  overall  ranking  of  the 
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officers.  The  final  CGSC  selections  are  based  upon  the  OML  and  are  dependent  upon  the 
number  of  seats  available  by  branch  specialty. 

Statement  of  the  Problem 

Due  to  its  downsized  force,  the  Army  can  ill  afford  to  have  a  large  number  of  officer 
experts  spending  significant  manhours  reviewing  personnel  records.  The  nature  and  process 
of  the  first  cut  of  the  school  selection  process  demonstrates  a  real  potential  for  the 
development  of  a  computerized  system  to  perform  the  expert  function  of  initial  record 
review. 

Problem  Statement 

The  purpose  of  the  study  was  to  begin  the  development  and  testing  of  an  Expert 
System  (ES)  to  screen  officer  personnel  records  being  considered  for  CGSC.  The  relatively 
short  time  allotted  for  human  decision-making  about  a  large  number  of  records  during  a 
selection  board  poses  a  time  problem  for  many  human  experts  but  would  not  for  an  expert 
system.  Specifically  an  ES  designed  to  utilize  a  scanning  mechanism  could  review 
significantly  more  data  elements  in  a  much  shorter  time.  It  is  theoretically  possible  that  an 
expert  system  could  be  designed  to  support  the  decision  making  process,  eliminate  one  step 
of  the  process,  but  not  replace  the  human  expert  in  the  final  stage  of  the  hard  vote. 

Research  Question 

The  following  research  question  was  addressed  in  this  study:  can  an  expert  system  be 
useful  to  accurately  select  individuals  for  the  hard  vote  during  the  CGSC  board  selection 
process? 
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During  the  conduct  of  this  study,  an  ES  was  considered  and  tested  as  a  support 
mechanism  for  the  human  decision  maker.  As  such,  the  system  was  designed  to  provide  the 
user  with  reasons  specific  records  were  rated  No,  allowing  for  manual  override  by  the  human 
experts. 

Strategic  Significance  of  the  Study 

At  the  United  States  Army  War  College  (USAWC),  strategic  leadership  is  defined  as 
"the  process  used  by  a  leader  to  affect  the  achievement  of  a  desirable  and  clearly  understood 
vision  by  influencing  organizational  culture,  allocating  resources,  directing  through  policy 
and  directives,  and  building  consensus"^.  Using  this  definition,  leaders  looking  at  the  school 
selection  process  would  be  considered  to  be  thinking  and  planning  strategically  if  they  risked 
changing  the  present  system  by  envisioning  a  new  way  to  allocate  and  make  use  of  limited 
resources  (board  members).  By  directing  new  policy  toward  the  development  and  utilization 
of  existing  high  technology  capabilities,  a  long  standing  decision  paradigm  could  be  reshaped 
and  improved. 


Review  of  the  Literature 

Understanding  the  process  of  developing  an  expert  system  is  dependent  upon  a  general 
exposure  to  the  field  of  artificial  intelligence  (AI).  Toward  this  goal,  a  review  of  the  relevant 
AI  literature  was  made  as  related  to  human  resource  management  (HRM).  This  review 
includes  definitions  of  artificial  intelligence  and  expert  systems,  descriptions  of  ES  uses  in  the 
military  and  in  HRM,  and  a  discussion  of  the  potential  uses  and  the  limitations  of  ES  in  the 
personnel  field. 
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Artificial  Intelligence  and  Expert  Systems 


Artificial  intelligence  (AI)  has  been  defined  in  numerous  ways  ranging  from  the  very 
technical  to  the  very  practical.  Chamiak  and  McDermott  define  AI  as  "the  study  of  mental 
faculties  through  the  use  of  computational  models"®.  Shortiiffe  describes  it  even  more 
practically  as  "the  intelligence  of  any  machine  that  performs  a  task  that  a  century  ago  would 
have  been  considered  a  uniquely  human  intellecmal  ability"’. 

In  general,  AI  researchers  seek  to  represent  human  knowledge  in  an  electronic  or 
computerized  environment.  More  specifically,  AI  is  a  sub-domain  within  the  computer 
science  field  which  relies  on  symbolic  reasoning  and  heuristics  (rules  of  thumb)  rather  than 
on  simple  mathematical  calculations  or  routine  data  retrieval.  An  obviously  complex  area  of 
science,  artificial  intelligence  has  been  employed  in  many  fields  such  as  mathematics, 
engineering,  theorem  proving,  and  robotics. 

The  artificial  intelligence  application  most  commonly  connected  with  decision-making 
is  referred  to  as  an  expert  system.  An  expert  system  is  "a  computing  system  which  embodies 
organized  knowledge  concerning  some  specific  area  of  human  expertise,  sufficient  to  perform 
as  a  skillful  and  cost-effective  consultant"*.  The  technology  behind  the  development  of 
expert  systems  is  known  as  knowledge  engineering;  expert  systems  have  been  described  as 
rule-based,  knowledge-based,  or  expert  consulting  systems.  Expert  systems  are  dependent 
upon  an  inference  engine  (a  mechanism  by  which  the  computer  manipulates  the  rules  which 
have  been  programmed  to  deduce  inferences  or  to  make  diagnoses)  and  a  knowledge  base. 

An  inference  engine  may  be  re-used  in  a  variety  of  systems,  however,  the  knowledge  base  of 
an  expert  system  is  usually  domain  specific  to  that  particular  application  only. 
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Expert  Systems  in  the  Military 


Expert  systems  are  being  explored  and  utilized  in  the  military  in  a  number  of  ways. 
Most  of  the  applications  relate  to  military  intelligence,  command  decision  making, 
instruction,  and  data-base  management®.  Other  smaller  systems  have  also  been  developed  to 
handle  HRM  issues  such  as  the  slating  procedures  for  the  Senior  Service  College 
fellowships.^®  The  impact  of  AI  and  ES  in  the  military  is  acknowledged  and  as  General 
Maxwell  Thurman  attested,  "[t]he  development  of  automated  support  [for  the  Army],  resting 
in  part  on  the  application  of  artificial  intelligence  and  related  techniques... is  essential  to  the 
successful  planning,  support,  and  operation  of  numerically  outnumbered  and  dispersed 
forces 

Expert  Systems  in  Human  Resource  Management 

Seaman  states,  "(t)he  major  purpose  of  expert  systems  is  to  help  people  make 
decisions  it  is  logical  therefore  that  many  expert  systems  have  been  developed  for  use 
specifically  in  the  HRM  field.  Byun  and  Suh‘^  provide  an  excellent  overview  of  the  ways  in 
which  ES  can  be  successful  in  assisting  managers  in  critical  decision  making.  They  suggest 
that  the  HRM  domains  of  planning,  job  analysis,  recruitment,  selection,  performance 
evaluation,  compensation,  training,  and  labor-management  relations  are  the  most  appropriate 
for  the  development  of  ES. 

Several  of  the  specific  systems  which  have  been  developed  in  the  HRM  arena  will  be 
described  in  the  following  paragraphs  and  include  the  Service  Selection  Advisor^*, 
Organizational  Consultants^,  Resumix}^,  and  EXPERS^,  While  not  an  inclusive  or  exhaustive 
listing  of  ES  available,  this  sample  of  existing  programs  introduces  the  utility  of  ES  in  HRM. 
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Service  Selection  Advisor  (SSE)  was  developed  by  researchers  at  Virginia  Polytechnic 
Institute  and  State  University  primarily  to  "support  the  career  field  choices  made  by  senior 
midshipmen  in  Naval  ROTC  programs"^*.  The  program  had  a  secondary  purpose  of 
decreasing  the  advising  load  of  the  ROTC  faculty.  SSE  assisted  midshipmen  in  choosing 
career  fields  by  analyzing  results  of  a  narrative  summary  of  the  student,  a  physical  exam,  and 
a  security  questionnaire  balanced  agaiost  six  branch  choices  of  the  student.  This  rule-based 
system  was  developed  using  archival  files  of  previous  selections  and  specific  mathematical 
equations  derived  from  the  factors  noted  above.  The  developers  were  pleased  with  the 
outcomes  of  the  ES  advising.  They  reported  that  the  ES  "performs  as  well  as  human 
advisors  in  evaluating  a  midshipman's  chances  for  being  selected  for  a  particular 
designator"*^. 

Organizational  Consultant  was  designed  to  assist  in  organizational  structure  decisions. 
Its  knowledge  base  of  250-plus  rules  represents  an  ES  attempt  to  match  an  appropriate 
organizational  structure  with  a  stated  or  given  organizational  environment.  The  system's 
decision  output  includes  information  regarding  the  structure,  formalization,  complexity,  and 
span  of  control  recommended  for  the  organization  under  consideration. 

Resumix  demonstrates  the  capability  of  an  ES  to  evaluate  a  resume  or  job  application. 
This  particular  system  is  connected  to  an  optical  scanner  and  facilitates  the  processing  of 
huge  numbers  of  resumes  at  a  very  fast  rate.  According  to  Booker^®,  this  system  saves  time 
in  the  review  of  applications  and  has  the  ability  to  keep  the  information  from  those 
documents  in  a  database  for  access,  retrieval,  and  evaluation  at  any  time.  The  system  was 
reported  to  have  been  used  by  the  Clinton  White  House  to  wade  through  volumes  of  resumes 
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for  positions  in  the  new  administration^*. 

One  final  example  of  a  working  ES  in  HRM  is  a  system  named  EXPER.  This  system 
was  developed  as  part  of  an  overall  human  resource  management  system  with  a  particular 
focus  on  accurate  and  appropriate  job  placement.  EXPER  evaluates  employees  in  order  to 
match  their  aptitudes  with  specific  jobs  within  an  organization.  Test  scores,  educational 
levels,  performance  ratings,  and  aptitude  scores  are  among  the  variables  entered  into  the 
database  and  considered  by  the  ES  to  assist  managers  in  making  placement  decisions. 

Potential  Uses  and  Limitations  of  ES  in  HRM 

As  more  and  more  firms  downsize,  the  field  of  HRM  appears  to  be  prime  for  the 
development  of  ES  applications.  At  the  same  time,  the  limitations  of  such  systems  must  be 
assumed  up  front  and  corrected  for  if  necessary  in  an  effort  to  keep  the  person  in  the 
personnel  department. 

A  long  laundry  list  of  capabilities  and  benefits  for  ES  can  be  easily  compiled. 
Increased  productivity,  consistent  performance,  and  institutionalized  expertise  are  among  the 
system  capabilities  embedded  into  specific  programs^.  ES  have  the  potential  to  assist  with  a 
wide  variety  of  tasks  including  matching  people  to  jobs,  planning  career  paths,  selecting  new 
employees,  and  training  new  and  old  employees^^.  In  addition,  the  capability  of  an  expert 
system  to  explain  its  decision  making  process  is  an  added  benefit  to  the  system  user.  ES 
generated  explanations  are  especially  useful  to  a  manager  who  is  first  and  foremost  assisted 
by  the  computer  in  the  decision  making  process  and  is  then  able  secondarily  to  learn  how  and 
why  a  particular  decision  was  made.  The  explanatory  mode  of  such  systems  also  works  as  a 
double-check  or  safeguard  and  permits  the  human  user  to  override  the  computer  decision 


9 


making  if  the  reasoning  is  inaccurate  or  subject  to  an  exception  to  policy. 

Nonetheless,  there  can  be  limitations  to  some  expert  systems.  Limitations  include 
incorrect  knowledge,  difficulty  in  obtaining  knowledge  from  the  appropriate  experts, 
difficulty  representing  that  knowledge  in  a  computer  model,  lack  of  the  computer's  ability  to 
learn,  and  a  normal  human  fear  of  the  computer  taking  over  the  decision  making  process^'*. 
Some  of  the  limitations  of  ES  can  be  overcome  by  proper  utilization  and  user  expectation. 
Systems  which  are  designed  to  be  support  systems  should  not  be  expected  to  be  otherwise.  In 
addition,  Illovslty  points  out  that  some  limitations  (such  as  ES  not  being  able  to  handle 
complex  cognitive  tasks  or  the  inability  of  the  systems  to  learn)  can  be  overcome  by  expert 
progranuning  and  system  maintenance^. 


Methodology 

This  study  was  conceived  as  a  result  of  personal  experiences  on  a  Department  of  the 
Army  Command  and  General  Staff  Selection  Board.  The  process,  at  least  the  first  vote, 
seemed  to  be  an  example  of  a  problem  well  suited  to  assistance  from  an  expert  system. 
Selecting  fully  qualified  records  appeared  to  meet  at  least  two  of  the  problem  selection 
criteria  set  forth  by  Mockler  and  Dologite^.  Those  criteria  were:  1)  the  human  expertise  was 
expensive  in  relation  to  the  job's  value;  and  2)  the  job  requirements  (such  as  speed  and 
precision)  exceeded  the  capacity  of  normally  available  experts.  One  general  officer,  six 
colonels,  and  three  lieutenant  colonels  comprised  the  board  membership  and  several  of  the 
members  struggled  to  keep  up  the  pace  of  the  Yes/No  record  review. 

An  ES  is  developed  according  to  a  logical  plan  which  includes  the  problem  selection, 
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knowledge  acquisition,  knowledge  representation,  knowledge  encoding,  knowledge  testing 
and  evaluation,  and  implementation  and  maintenance^’.  The  following  sequence  of  events  led 
to  the  initial  development  of  the  ES  named  Board  Expert  (BOARDEX). 

Knowledge  acquisition  is  the  process  of  acquiring  and  organizing  the  knowledge  to  be 
used  in  a  knowledge-based  system.  In  this  instance,  knowledge  acquisition  was  a 
combination  of  documenting  specific  criteria  for  selection  outlined  in  Army  Regulations  and 
coupling  them  with  the  important  elements  considered  when  reviewing  an  officer's  record  as 
defined  by  a  domain  expert.  The  investigator  became  the  domain  expert  based  upon  her 
board  experience.  She  enumerated  specific  data  necessary  during  the  board  record  review 
and  the  hierarchical  relationships  between  elements  were  established.  Questions  were 
constructed  and  decision  matrices  based  upon  binary  answers  to  them  were  generated.  Table 
1  represents  the  factors  considered. 

Knowledge  representation  was  accomplished  with  the  help  of  CPT  Thomas  Galvin,  a 
staff  member  from  the  Knowledge  Engineering  Group,  Center  for  Strategic  Leadership, 
United  States  Army  War  College.  CPT  Galvin  suggested  using  PROLOG^  as  the  basis  for 
this  system.  He  pointed  out  that  PROLOG  is  particularly  effective  when  the  decision  process 
is  primarily  binary.  Questions  and  the  hierarchical  relationships  between  them  were  refined 
on  several  occasions.  Decision  matrices  were  devised.  Failure  to  meet  hard  and  fast 
(regulatory)  criteria  for  selection  yielded  immediate  non-selection  decision;  failure  to  meet 
other  discretionary  factors  yielded  a  maybe  response.  Maybe  responses  required  linking  and 
qualification  by  other  data  elements.  Very  simple  heuristics  regarding  awards,  deployment 
history,  and  job  descriptions  were  devised  but  proved  to  be  the  most  difficult  procedure. 
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Ultimately  this  process  was  accomplished  by  defining  which  data  elements  constimte  a  plus 
or  a  minus  for  a  record.  A  percentage  of  possible  awards  and/or  earned  badges  documented 
in  the  record  was  then  established  as  a  cut  off-point. 


Grade 

Y/N 

=  or  >  CPT  (P)? 

Active  Federal 

Commissioned  Service  (AFSC) 

y/N 

<168  months? 

Military  Education  Level  (MEL) 

Y/N 

»  or  <  6? 

Photograph 

R 

Recent  (<  2  years)? 

C 

Current  (>2  and  <  5  years)? 

0 

Outdated  (>  5  years)? 

Height/We ight 

Y/N 

Meets  AR  600-9  standards? 

Assignments 

Y/N 

Varied? 

Y/N 

Command  or  branch  qualified? 

OER  Part  TV 

Y/N 

All  Is  on  last  four  OERs? 

Performance 

Y/N 

Left  block  on  last  four  OERs? 

Potential 

Y/N 

Left  block  on  last  four  OERs? 

Senior  Rater 

Y/N 

Top  block  on  last  four  OERs? 

Civilian  Education  Level  (CEL)  I 

Y/N 

=  or  <  2?  1 

Table  1.  Factors  Considered 


Knowledge  encoding  is  the  transfer  of  information  to  the  language  of  the  expert 
system.  This  was  accomplished  by  entering  data  from  a  set  of  15  mock  board  files  obtained 
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from  the  Department  of  the  Army  Secretariat,  US  Total  Army  Personnel  Command 
(PERSCOM).  Each  record  was  reviewed  individually  and  responses  to  the  questions  in  Table 
1  were  entered  into  the  PROLOG  program.  Awards,  job  descriptions,  and  deployments  were 
also  entered  into  the  ES.  During  the  process,  the  order  of  questions  was  revised  to  assist 
with  more  rapid  data  entry. 

The  next  step  in  the  process  of  system  development  is  the  knowledge  testing  and 
evaluation.  This  was  a  two-part  strategy  and  included  a  comparison  between  the  selections 
by  the  expert  system  and  a  human  expert  board.  In  essence,  the  human  experts  were  used  to 
validate  the  selections  by  BOARDEX. 

Thirteen  members  of  the  USAWC  Class  of  1996  reviewed  the  records  and  voted  them 
as  if  they  were  selection  board  members.  This  group  was  purely  a  convenience  sample  and 
did  not  reflect  any  attempt  to  provide  a  random  sample  for  statistical  purposes.  Because 
three  of  the  13  were  not  U.S.  Army  officers  (voter  numbers  8,  10,  and  12),  they  were 
eliminated  from  the  analysis.  Of  the  ten  remaining  board  members,  only  one  had  previous 
board  experience,  two  had  mock  board  experience,  and  the  other  seven  had  no  board 
experience.  The  board  members  were  briefed  as  to  the  voting  process,  the  composition  of 

t 

the  records,  and  counseled  as  to  a  limited  amount  of  time  to  be  spent  on  each  record;  it  was 
recommended  that  they  spend  no  more  than  three  minutes  on  each  record.  Votes  were  tallied 
and  results  documented.  Candidates  were  considered  to  be  selected  if  they  received  Yes  votes 
from  at  least  five  (50%)  of  the  board  members. 

The  second  part  of  the  testing  and  evaluation  phase  was  accomplished  by  actually 
running  the  ES.  The  selections  and  comparisons  between  the  ES  and  the  human  expert  board 
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will  be  provided  in  the  results  section  of  this  paper. 

The  final  phase  of  the  development  of  an  ES  is  implementation  and  maintenance.  Due 
to  the  time  constraints  of  the  strategic  research  project,  these  two  functions  were  well  beyond 
the  scope  of  this  paper.  In  theory,  however,  in  the  next  step  of  development,  the  ES  would 
be  implemented  using  a  user  friendly  interface  rather  than  the  complex  PROLOG  language 
and  adjustments,  corrections,  and  changes  would  be  made  to  the  rules  and  heuristics  based 
upon  changes  in  policy  and  practice. 


RESULTS 

The  results  of  this  study  can  be  described  as  the  analysis  of  the  correlations  between 
two  separate  sets  of  data.  First,  the  Pearson  correlations  were  calculated  between  the  voters 
on  the  human  expert  panel.  Second,  correlations  were  determined  between  the  votes 
generated  by  BOARDEX  and  the  human  expert  board.  All  statistics  were  calculated  using 
the  Statistical  Package  for  the  Social  Sciences  (SPSS  6.1  for  Windows). 

The  first  analysis  yielded  a  large  number  of  statistically  significant  correlations 
between  individual  board  members.  Raw  voting  data  of  both  BOARDEX  and  the  10  human 
experts  is  available  at  Appendix  B.  Appendix  C  is  a  correlation  matrix  based  upon  the 
human  expert  data.  Twenty-six  of  the  possible  45  correlations  between  voters  were 
statistically  significant  at  p  <  or  =.05  level  of  significance. 

The  second  analysis  was  more  complex.  BOARDEX  was  developed  with  a  capability 
of  determining  three  categories  of  record:  those  which  were  strongly  recommended  for  hard 
vote;  those  which  were  moderately  recommended  for  hard  vote;  and  those  which  were 
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recommended  for  immediate  nonselection.  In  everyday  terms,  these  translate  into  Yes, 
Maybe,  and  No.  In  the  board  setting,  the  Maybe  votes  will  most  likely  become  Yes  votes 
because  voters  will  want  to  review  that  file  again  in  more  detail  during  the  hard  vote.  The 
votes  were  therefore  correlated  in  two  ways,  straight  Yes/No,  and  Yes/Maybe/No.  The 
Pearson  correlation  for  the  Yes/No  vote  was  r=.6614  (p=.007)  and  for  the  Yes/Maybe/No 
was  r=.6646  (p=.007)  recoding  the  human  expert  votes  by  the  number  of  Yes  votes  per 
record.  A  record  was  coded  No  if  0-4  board  members  voted  Yes,  Maybe  if  5-8  members 
voted  Yes,  and  Yes  if  9-10  members  voted  Yes.  In  addition,  cross-tabs  were  run  to  determine 
an  evaluation  concordance  and  are  found  in  Tables  2  and  3.  Each  of  the  evaluation 
concordances  were  found  to  be  statistically  significant  using  both  Pearson  and  the  Spearman 
correlation. 
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Count 

Row  Percent 

Column  Percent 

Total  Percent 

HUMAN  EXPERTS 

NO 

HUMAN  EXPERTS 

YES 

5 

3 

BOARDEX 

62.5 

37.5 

NO 

100.0 

30 

33.3 

20.0 

7 

BOARDEX 

100.0 

YES 

70.0 

46.7 

Statistic 

Value 

Approximate  Significance 

Pearson's  R 

.66144 

.00724 

Spearman  Correlation 

.66144 

.00724 

*  Significance  is  based  on  a  normal  approximation 

Table  2.  Evaluation  Concordance  for  Yes/No  votes. 
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Count 

Row  Percent 

Column  Percent 

Total  Percent 

HUMAN 

EXPERTS 

NO 

HUMAN 

EXPERTS 

MAYBE 

HUMAN 

EXPERTS 

YES 

5 

2 

1 

BOARDEX 

62.5 

25.0 

12.5 

NO 

100.0 

50.0 

16.7 

33.3 

13.3 

6.7 

1 

2 

BOARDEX 

33.3 

66.7 

MAYBE 

25.0 

33.3 

6.7 

13.3 

1 

3 

BOARDEX 

25.0 

75.0 

YES 

25.0 

50.0 

6.7 

20.0 

Statistic 

Value 

Approximate  Significance* 

Pearson's  R 

.66463 

.00687 

Spearman  Correlation 

.66484 

.00485 

*  Significance  is  based  on  a  normal  approximation 

Table  3.  Evaluation  Concordance  for  Yes/Maybe/No  votes. 
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DISCUSSION,  CONCLUSION,  AND  RECOMMENDATIONS 

This  study  demonstrated  the  utility  of  a  very  simple  expert  system  for  use  in  human 
resource  management.  Specifically,  BOARDEX  was  successful  at  selecting  those  records 
dining  a  mock  CGSC  board  which  would  be  recommended  for  further  consideration. 

Highly  significant  correlations  and  evaluation  concordances  between  the  system  and  the 
human  experts  were  encouraging  and  the  process  validated  the  effectiveness  and  efficiency  of 
ES  in  HRM. 

The  number  of  significant  correlations  between  members  of  the  human  expert  board 
demonstrated  a  high  internal  consistency  for  the  voting  procedure.  While  there  were  some 
outliers,  in  general  the  voting  was  relatively  uniform.  The  Yes  versus  No  records  were  easily 
determined  by  the  voting  patterns  as  many  records  were  either  unanimously  Yes  or 
unanimously  No. 

The  significant  findings  between  BOARDEX  and  the  human  expert  board  are 
encouraging.  The  evaluation  concordance  demonstrated  a  consistency  between  the  two 
modes  of  voting  that  could  be  improved  with  further  refinement  of  the  system.  Intellectually, 
few  board  members  would  have  trouble  if  a  computer  model  were  to  select  a  record  for 
continued  consideration;  the  difficulty  in  logic  and  emotion  enters  the  picture  if  a  computer 
were  to  mechanically  non-select  individuals  thereby  eliminating  them  from  contention.  One  of 
the  advantages  of  BOARDEX  is  that  it  can  explain  the  reason(s)  that  someone  is  non-select. 
This  explanatory  mode  allows  the  board  to  review  any  records  it  feels  were  rejected  for 
inappropriate  or  nonapplicable  reasons.  The  system  can  also  be  prompted  to  enumerated 
those  records  which  are  found  to  be  either  sufficient  or  insufficient  in  a  particular  area.  For 


18 


instance,  the  system  is  able  to  list  all  those  individual  records  whose  photograph  is  outdated 
or  missing. 

Limitations  of  the  Study 

This  study  was  limited  by  several  factors  including  the  limitation  of  time  for  testing 
and  the  small  number  of  records  obtained  from  PERSCOM  for  review.  Other  limitations 
were  evident  regarding  the  investigator's  amateur  ability  to  develop  and  manipulate  the  ES 
programming  and  language. 

Although  the  inter-rater  correlations  between  the  board  members  was  good,  factors 
that  may  have  biased  the  board  included:  the  nature  of  the  board  scenario  was  contrived 
because  the  records  reviewed  were  actually  a  Major  to  Lieutenant  Colonel  promotion  board 
set  not  actual  CGSC  records;  no  quota  or  goal  for  selection  was  established;  and  the  board 
members  did  not  go  through  a  calibration  of  their  voting.  A  confounding  factor  in  the  data 
entry  of  the  ES  was  that  the  photographs  were  totally  out  of  date.  They  ranged  in  date  from 
1988  through  1991  and  were  totally  unacceptable  with  regard  to  the  standards  established  in 
the  decision  matrix  and  current  doctrine.  This  was  corrected  in  BOARDEX  by  establishing 
specific  years  for  the  categories  recent,  current,  and  outdated,  but  this  was  a  variance  from 
the  original  design  of  the  ES.  A  method  to  concurrently  test  an  ES  with  actual,  current 
records  would  be  of  value. 

Recommendations 

After  working  with  this  system  and  talking  with  experts  in  the  field  of  AI,  it  is 
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obvious  to  the  investigator  that  BOARDEX  is  really  not  an  ES  but  truly  at  the  level  of  a 
decision  support  system.  Several  recommendations  for  further  work  and  research  could  help 
to  enhance  the  system  and  to  increase  not  only  its  decision  capability  but  also  its  usefulness  in 
military  HRM. 

First,  a  user  interface  for  the  system  should  be  developed.  If  BOARDEX  is  to  be 
helpful  in  cutting  down  the  use  of  high  ranking  manpower  to  screen  records  prior  to  a  hard 
vote,  the  access  to  the  program  must  be  logical,  user  friendly,  and  relatively  fast.  Ideally  the 
system  could  interface  with  the  existing  PERSCOM  data  base  which  generates  the  ORB.  An 
additional  advantage  would  be  to  couple  the  ES  with  an  optical  scanner  as  was  done  with 
Resumix,  to  read  the  OERs  and  ORBs  and  speed  up  data  entry. 

Second,  a  better  system  would  include  more  sophisticated  heuristics  to  differentiate 
between  records.  In  order  to  accomplish  this,  a  panel  of  experts  could  be  surveyed  and  a 
decision  hierarchy  agreed  upon.  In  addition,  a  more  sophisticated  decision  system  would  be 
able  to  analyze  the  relationship  and  timing  of  important  criteria  and  weigh  the  relative  merit 
of  specific  senior  raters. 

A  final  recommendation  is  to  revise  the  system  until  it  is  detailed  and  discriminating 
enough  to  provide  a  computer  scan  of  the  records  for  the  hard  vote.  This  would  save  time 
for  the  board  members  who  must  juggle  the  microfiche  reader  and  hard  copies  of  documents 
in  order  to  make  an  informed  decision. 
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APPENDK  A 


PROLOG  Source  Code  Listing 


run  :  - 

bagof (X, select (X) , SelectList) , 

write ('The  following  are  strongly  recommended  for  hard 
vote : ' ) ,  nl, 

for_each_write (SelectList) , 
bagof (n (X, L) , nonselect (X, L) ,NonSelectList) , 
write ('The  following  are  recommended  for  immediate 
nonselection:'),  nl, 

for_each_write (NonSelectList ) , 
bagof (n(X,L) , maybe (X,L) ,MaybeList) , 

write ('The  following  are  moderately  recommended  for  hard 
vote : ' ) ,  nl, 

for_each_write (MaybeList) . 

select (X) 

candidate (X) , 

meets_immediate_screening_requirements  (X) , 
meets_photo_standards (X) , 
meets_height_weight_standards (X) , 
not (meets_ar_600_9_via_body_taping (X) ) , 
meets_assignment_standards (X) , 
no_bad_oers (X) . 

nonselect (X, ReasonL)  : - 
candidate (X) , 

bagof (HL, nonselect 1 (X, HL, hell_no) ,L1) ,  nice_append (LI, LIA) , 
bagof (ML, nonselectl (X, ML, maybe) ,L2) ,  nice_append(L2,L2A) , 
append (LIA, L2A, L3) ,  nonselectO (X, LIA, L3, ReasonL) . 
nonselectO (_, LIA, ReasonL, ReasonL)  not (same (LIA, [])) ,  !. 

nonselectO (X, [ ] , ReasonL, [plusses_dont_of f set | ReasonL] )  : - 

not (plusses_of f set_all_nonselect_reasons (X, ReasonL) ) ,  ! . 

nonselectl (X, ReasonL, hell_no) 

fails_immediate_screening_requirements (X, ReasonL) . 
nonselectl (X, outdated_photo, maybe) 
fails_photo_standards (X) . 
nonselectl (X, heigh t_weight,hell_no) 

not (meets_height_weight_standards (X) ) . 
nonselectl (X, assignments, hell_no)  : - 
fails_assignment_standards (X) . 
nonselectl (X,multiple_bad_oers (OERL) ,hell_no) 
setof (bad_oer(P,OER) ,bad_oer (X, P,OER) ,OERL) , 
same  (OERL,  [_/_!_]  )  • 

maybe (X, ReasonL) 
candidate (X) , 
not (select (X) ) , 
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not (nonselect (X,_) ) , 

bagof (HL, nonselectl (X, HL, hell_no) , [ ] ) , 

bagof (ML, nonselectl (X, ML, maybe) ,L1) ,  nice_append (LI, LIA) , 
bagof (NL,maybel (X,NL) ,L2) ,  nice_append (L2,L2A) , 
append (LIA, L2A, L3) ,  maybeO (X, LlA, L3, ReasonL) . 
maybeO (_, [] ,ReasonL,ReasonL)  !. 
maybeO (X, LIA, ReasonL, [plusses_of f set | ReasonL] )  : - 

not (same (LIA, [] ) ) ,  plusses_of fset_all_nonselect_reasons (X,L3) , 

I  ^ 

maybel (X,body_fat) 

meets_height_weight_standards (X) , 
meet s_ar_600_9_via_body_t aping  (X)  . 
maybel (X, ReasonL) 

maybe_assignment_standards (X, ReasonL) . 
maybel (X, one_bad_oer (bad_oer (P, OER) ) ) 
setof  (OER,  P''bad_oer  (X,  P,  OER)  ,[_]), 
bad_oer (X,P,OER) . 

meets_immediate_screening_requirements (X) 
promotion_eligible_captain (X) , 
afsc(X,AFSCl) ,  AFSC1<230, 
mel (X,MEL1) ,  greater_than_MEL_6 (MELl) . 
meets_immediate_screening_requirements (X) 
major  (X)  , 

afsc(X,AFSCl) ,  AFSC1<230, 

mel (X,MEL1) ,  greater_than_MEL_6 (MELl) . 

greater_than_MEL_6 (X)  : -  member (X, [n, 6, 4, 2] ) . 

fails_iinmediate_screening_requirements  (X,  ineligible) 
not (promotion_eligible_captain (X) ) ,  not (ma j  or (X) ) . 
fails_immediate_screening_requirements (X, beyond_window)  : - 
afsc(X,AFSCl) ,  AFSC1>=230. 

fails_immediate_screening_requirements (X, education)  : - 
mel (X,MEL1) ,  not (greater_than_MEL_6 (MELl) ) . 

meets_photo_standards (X) 
photo (X, recent ) . 
raeets_photo_standards (X) 
photo (X, current) . 
fails_photo_standards (X) 
not (photo (X,_)  ) . 
fails_photo_standards (X) 
photo (X, outdated) . 

/*  meets_height_weight_standards  is  a  database  fact  */ 

meets_assignment_standards (X)  :  - 

has_varied_assignments (X) , 
branch_qualified (X) . 
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fails_assigninent_standards  (X)  :  - 

not (has_varied_assignments (X) ) , 
not {branch_qualif ied (X) ) . 

maybe_assignment_standards (X, varied)  : - 
branch_qualif ied (X) , 
not  (has_varied_assigninents  (X)  )  . 
maybe_assignment_standards (X,branch_qual) 
has_varied_assignments (X) , 
not (branch_qualif ied (X) ) . 

no_bad_oers (X)  not (bad_oer (X,_,_) ) ,  !  . 

no_bad_oers (X)  : -  bad_oer (X, P, BOER) ,  is_an_oer (X, GOER) , 

not (same (BOER, GOER) ) , 

of f set_bad_oer (X, GOER, BOER)  ,  ! . 

good_oer (X, OER)  above_center_of_mass_OER (X, OER) . 
good_oer (X, OER)  at_center_of_mass_OER (X, OER)  . 

bad_oer (X, performance, OER)  performance_rating (X, OER,  PERF) , 
not (same (PERF, left_column) ) . 

bad_oer (X, potential, OER)  potential_rating (X, OER,  POTN) , 

not (same (POTN, left_column) ) . 

bad_oer (X,part_IV,OER)  part_IV_rating (X, OER,  PTIV)  , 

not (same (PTIV, all_ls) ) . 
bad_oer (X,below_center_of_mass, OER) 
senior_rating (X, OER, RATING)  , 
senior_rater (X, OER, SRATER, RANK) , 
below_center_of_mass (RATING, SR_RATER) . 

offset_bad_oer (X,OERl,OER2) 

good_oer (X, OERl) ,  senior_rater (X, OERl, SRATERl, go) , 
senior_rater (X,OER2, SRATER2 , not_go ) . 
offset_bad_oer (X, OERl, OER2) 

good_oer (X, OERl ) ,  senior_rater (X, OERl , SRATER, _) , 
senior_rater (X, OER2 , SRATER, _) , 
more_recent_than(OERl,OER2) . 

plusses_of fset_all_nonselect_reasons  (_,  [] )  . 
plusses_of fset_all_nonselect_reasons (X, [R|RL]) 
plusses_of f set_nonselect (X,  R)  , 
plusses_of f set_all_nonselect_reasons (X,RL) . 
plusses_of f set_all_nonselect_reasons (_,_)  !,  fail. 

plusses_of f set_nonselect (X, outdated_photo) 
plusses (X, N) ,  N  >=  4 . 
plusses_of f set_nonselect (X,part_III (_) ) 
plusses (X, N) ,  N  >=  3 . 
plusses_of f set_nonselect (X,part_IV (_) ) 
plusses (X, N) ,  N  >=  3. 
plusses_of f set_nonselect (X,_)  !,  fail. 
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plusses(X,N)  bagof (P, has_a_plus (X, P) , PL) ,  length (PL, N) . 

has_a_plus (X, cel)  cei(X,l). 

has_a_plus (X, cel)  cel{X,2). 

has_a_plus (X, cel)  cel(X,3). 

ha s_a_plus (X, badge 1)  bagof (B, earned (X, B) , [_|_] ) . 

has_a_plus (X,badge2)  bagof (B, earned (X, B) , . 
has_a_plus (X, deployment_experience) 
has_deployinent_experience  (X,  W)  . 

has_a_plus (X, general_aide)  duty_title (X, OER, general_aide) . 
has_a_plus (X, pentagon_badge)  award (X,pentagon_badge) . 

for_each_write ( [] ) . 

for_each_write ( [X I L] )  atomic (X),  write ( '  write (X),  nl, 

for_each_write (L) . 

for_each_write ( [n (X, XL) I L] )  write ( '  ')f  write (X),  write (' , 

because: ' ) ,  nl, 

for_each_writel {X,XL) ,  for_each_write (L) . 
for_each_writel  (_,[]). 

for_each_writel (X, [plusses_of fset I L] )  write ( '  '), 

explanation (plusses_off set, XE) ,  write (XE),  nl, 
for_each_writel (X,L) , 

write ( '  The  following  were  the  offsetting  items:'),  nl, 
write_plusses (X) . 

for_each_writel (X, [E I L] )  :-  explanation (E, '') , 

for_each_writel (X, L) . 

for_each_writel (X, [E I L] )  :-  explanation (E,XE) ,  not (same (XE, '')) , 

write ( '  '),  write (XE),  nl,  f or_each_writel (X, L) . 

write_plusses (X)  :-  setof (B, has_a_plus (X, B) , BL) , 

( (member (badge2, BL) , delete (badgel, BL, BLL) ) ; same (BL, BLL) ) , 
write_plussesl (BLL) . 
write_plussesl ( [] ) . 

write_plussesl ( [B I BL] )  :-  write ( '  +  '),  explanation (B, BE) , 

write (BE),  nl, 

write_plussesl (BL) . 


explanation (plusses_dont_of fset ,' The  officer' 's  special 
qualifications  did  not  offset  the  below  problems.'), 
explanation (plusses_of fset, ' The  officer' 's  special  qualifications 
offset  the  below  problems.'). 

explanation (ineligible, ' Of ficer  is  not  a  promotion-eligible 
captain. ' )  . 

explanation (beyond_window, ' Off icer  has  already  been  considered 
four  times . ' ) . 

explanation (education, ' Officer  has  not  met  military  education 
requirements. ' ) . 

explanation (outdated_photo, ' Photo  is  over  five  years  old.'), 
explanation (height_weight, ' Fails  to  meet  height  and  weight 
standards . ' ) . 

explanation (assignments, ' Has  not  met  branch  requirements  for 
assignments . ' ) . 
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explanation (varied, ' Has  not  sufficiently  varied  assignments.'), 
explanation (branch_qual, ' Has  not  held  command  or  other  branch 
qualifying  position. ' ) . 

explanation (body_f at, ' Required  taping  to  meet  AR  600-9 
standards . ' ) . 

explanation (multiple_bad_oers (OERL) , ' ' )  : - 

write ( '  Officer  has  more  than  one  weak  OER  among  last 
four. ' ) ,  nl, 

write_plussesl (OERL) . 
explanation (one_bad_oer (OER)  ,  ' ' ) 

write  ( '  Officer  has  one  weak  OER  among  last  four.'),  nl, 
write_plussesl ( [OER] )  . 
explanation (bad_oer (performance,  OER) ,  ' ' )  : - 

write ('The  performance  block  of  OER  #'),  write (OER), 
write  ( '  is  checked  away  from  the  left  column.'), 
explanation (bad_oer (potential , OER) , ' ' )  : - 

write  ('The  potential  block  of  OER  #'),  write (OER), 
write ( '  is  checked  away  from  the  left  column.'), 
explanation (bad_oer (part_IV, OER)  : - 

write ('OER  #'),  write (OER) ,  write ('has  non-1  ratings  on  the 
front  side. ' ) . 

explanation (bad_oer (below_center_of_mass, OER) , ' ' ) 

write ('The  senior  rater  rated  the  officer  below  center  of  mass 
on  OER  # ' ) , 

write (OER) ,  write ('.'). 
explanation (cel, 'Civilian  education  level'). 

explanation (badgel, ' Earned  one  special  qualification  badge  (ABN, 
SF,  AASLT,  etc. )  ' )  . 

explanation (badge2, ' Earned  two  special  qualification  badges  (ABN, 
SF,  AASLT,  etc. ) ' )  . 

explanation (deployment_experience, ' Has  deployment  experience'), 
explanation (generals_aide, ' Served  as  a  General' 's  Aide-de-Camp ' ) . 
explanation (pentagon_badge, ' Earned  the  Pentagon  Badge.'). 


goof (X,missing_mel, 'mel (<name>, <mel>) ') 
candidate (X) ,  not (mel (X,_) ) . 
goof (X,missing_rank, ' captain (<name>) , 

promotion_eligible_captain (<name>) ,  or  major (<name>) ' ) 
candidate (X) ,  not  (captain (X) ) , 
not (promotion_eligible_captain (X) ) , 
not (major (X) ) . 

goof (X,missing_afsc, 'afsc (<name>,<#  of  months  in  service>') 
candidate (X) ,  not (afsc (X,_) ) . 

goof (X,bad_af sc, 'afsc (<name>,<#  of  months  in  service  (numeric)>') 

candidate (X) ,  afsc(X,Y),  not (number (Y) ) . 
goof (X, bad_photo_status, ' photo (<name>,  recent  or  current  or 
outdated) 'T 

candidate (X) ,  photo (X,Y),  not (same (Y, current) ) , 
not (same (Y, recent) ) , 
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not (same (Y, outdated) ) . 

goof (X,missing_oer_data (Y) , ' duty_title (<name>,  <oer>,  <title>) ') 

candidate (X) ,  is_an_oer (X, Y) ,  not (duty_title (X, Y,_) ) . 
goof (X,missing_oer_data (Y) , ' performance_rating (<name>,  <oer>, 
<rating>) ' ) 

candidate (X) ,  is_an_oer (X, Y) ,  not (performance_rating (X, Y,_) )  . 
goof (X,bad_oer_data (Y) , 'performance_rating (<name>>  <oer>, 
<rating>) ' )  : - 

candidate (X) ,  is_an_oer (X, Y) ,  performance_rating (X, Y, P) , 

not (member (P, [left_column, second_column, third_column, right_column 
]))  • 

goof (X,missing_oer_data (Y) , 'potential_rating (<name>,  <oer>, 
<rating>) ' ) 

candidate (X) ,  is_an_oer (X, Y) ,  not (potential_rating (X, Y,_) ) . 
goof (X,bad_oer_data (Y) , 'potential_rating (<name>,  <oer>, 

<rating>) ' )  : - 

candidate (X) ,  is_an_oer (X, Y) ,  potential_rating (X, Y, P) , 

not (member (P, [left_column, second_column, third_column, right_column 

]))  . 

goof (X,missing_oer_data (Y) , 'part_IV_rating (<name>,  <oer>, 
<rating>) ' )  : - 

candidate (X) ,  is_an_oer (X, Y) ,  not (part_IV_rating (X, Y,_) ) . 
goof (X,bad_oer_data (Y) , 'part_IV_rating (<name>,  <oer>,  <rating>) ') 

candidate (X) ,  is_an_oer (X, Y) ,  part_IV_rating (X, Y, P) , 
not (member (P, [all_ls, some_2s] ) ) . 
goof (X,missing_oer_data (Y) , ' senior_rating (<name>,  <oer>, 

<rating>) ' )  : - 

candidate (X) ,  is_an_oer (X, Y) ,  not (senior_rating (X, Y,_) ) . 
goof (X, bad_oer_data (Y) , ' senior_rating (<name>,  <oer>,  <rating>) ' ) 

candidate (X) ,  is_an_oer (X, Y) ,  senior_rating (X, Y, P) , 

not  (member  (P,  [top_block;,  second_block,  third_block,  fourth_block,  fif 
th_block] ) )  . 

goof (X,missing_oer_data (Y) , ' senior_rater (<name>,  <oer>,  <rater>, 
<rank>) ' )  : - 

candidate (X) ,  is_an_oer (X, Y) ,  not (senior_rater (X, Y,_,_) ) . 
goof (X,bad_oer_data (Y) , ' senior_rater  (<name>,  <oer>,  <sr  rater>, 
<rank>) ' ) 

candidate (X) ,  is_an_oer (X, Y) ,  senior_rater (X, Y, SR, RK) , 
not (member (RK, [go,not_go] ) ) . 

goof (X,missing_sr_data (SR) , ' center_of_mass (<sr  rater>,  <block>) ' ) 

candidate (X) ,  is_an_oer (X, Y) ,  senior_rater (X, Y, SR,_) , 
not (center_of_mass (SR,_) ) . 

goof (X,bad_sr_data (SR) ,  ' center_of_mass (<sr  rater>,  <block>) ' ) 

candidate (X) ,  is  an  oer(X,Y),  senior_rater (X, Y, SR,_)  , 
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center  of  mass (SR, RK) 


not (member (RK, [top_block, second_block, third_block, fourth_block, fi 
fth  block] ) ) . 


is_an_oer (X, Y)  candidate (X) , 
setof (OER, is_an_oerl (X, OER) , OERL) , 
member ( Y, OERL) 
is  an  oerl(X,Y) 


is_an_oerl (X, Y) 
is_an_oerl (X, Y) 
is_an_oerl (X, Y) 
is_an_oerl (X, Y) 
is  an  oerl (X, Y) 


duty_title (X, Y,_) . 
performance_rating (X, Y,_) 
potential_rating (X, Y,_) . 
part_IV_rating (X, Y,_) . 
senior_rating (X, Y,_) . 
senior_rater (X, Y,_,_)  . 


append ( [ ] , L, L) . 

append( [X|M] ,L, [X|L1] )  append (M, L, LI) . 
delete  (X,  [],[])  . 
delete (X, [X|L] ,L) . 

delete(X, [Y|L] , [Y|L1] )  delete (X, L, LI) . 
member (X,L)  append (_, [X |_] , L) . 
nice_append ([],[] )  !. 

nice_append ( [X 1 L] ,  [X  I L3] )  not (is_list (X) ) ,  nice_append (L, L3)  , 

1  . 

nice_append( [L|L2] ,L3)  is_list(L),  nice_append (L, L5)  , 
nice_append(L2,L6) , 
append (L5, L6, L3)  ,  !. 

is_list ( [ ] )  . 
is_list  ([_!_])  . 
length ( [] , 0)  . 

length ( [XI L] ,N)  length (L, Nl) ,  N  is  Nl+1. 
same (X,X) . 
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APPENDIX  B 
Vote  Data 
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APPENDIX  C 


Pearson  Correlation  Matrix 
Human  Expert  Votes 


*  p  <  or  =  .05 
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